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1.  SUMMARY 


Autonomous  technologies  that  support  unmanned  vehicle  (UxV)  operations  are  typically 
developed  and  researched  in  isolation.  The  goal  of  the  Intelligent  Multi-UxV  Planner  with 
Adaptive  Collaborative/Control  Technologies  (IMPACT)  research  effort  is  to  integrate 
cooperative  control  algorithms  (CCAs)  that  optimize  route  planning,  intelligent  agents  (IAs)  that 
recommend  optimal  courses  of  action,  and  an  autonomies  framework  that  can  monitor  plans 
against  situations  to  help  an  operator  manage  multiple  heterogeneous  UxVs  in  dynamic  operating 
environments.  IMPACT  is  a  three  year  effort  that  seeks  to  combine  these  autonomous 
technologies  with  innovative  human-autonomy  interface  (HAI)  concepts  that  allow  operators  to 
effectively  supervise  multiple  UxVs  via  high  level  commands  called  plays.  For  example,  when 
an  IMPACT  operator  calls  a  play  to  obtain  air  surveillance  on  a  building,  the  IA  recommends  a 
UxV  to  use  (based  on  estimated  time  enroute,  fuel  use,  environmental  conditions,  etc.),  the  CCA 
provides  the  shortest  route  to  get  to  the  building  (taking  into  account  no-fly  zones),  and  the 
autonomies  framework  monitors  the  play’s  ongoing  status  (will  the  UxV  arrive  at  the  building  on 
time). 

This  report  describes  an  evaluation  of  IMPACT’S  Spiral  1  system.  Three  UxV  operators  and 
four  security  force  personnel  were  trained  on  IMPACT’S  autonomous  technologies  and  asked  to 
manage  six  UxVs  (three  unmanned  air  vehicles,  two  unmanned  ground  vehicles,  and  one 
unmanned  surface  vehicle)  in  support  of  a  simulated  base  defense  mission.  Feedback  was  sought 
on  the  HAI’s  candidate  display  formats,  symbology,  and  input  modalities  (mouse,  touchscreen, 
and  speech  recognition)  as  well  as  the  CCA,  IA,  and  autonomies  framework.  Subjective  data 
were  recorded  via  questionnaires  and  analyzed. 

Participant  feedback  was  uniformly  positive  in  regards  to  IMPACT’S  potential  value  to  aid  future 
UxV  operations,  to  reduce  operator  workload,  and  to  increase  operator  situation  awareness. 
Participants  also  rated  the  autonomous  technologies  and  HAI  favorably.  Additional  data  were 
collected  on  the  modality  participants  used  to  call  plays.  Participants  made  play  calls  using  the 
mouse  more  frequently  than  the  touchscreen  or  speech  commands.  Play  calling  with  the  mouse 
was  also  faster  and  less  error  prone  than  calling  plays  with  either  the  touchscreen  or  speech 
commands. 

These  findings  will  be  used  to  help  refine  and  demonstrate  effective  human-automation  interface 
and  interaction  technology  for  future  UxV  control  employment  concepts.  Specifically  for 
IMPACT,  these  findings  have  led  to  planned  improvements  for  IMPACT’S  Spiral  2  system, 
including  map-based  play  calling,  map  decluttering,  and  graphical  “widgets”  to  modify  play 
details  like  loiter  sizes.  Participant  feedback  also  led  to  additional  research  studies  focused  on 
how  to  best  organize  play  calling  symbology  as  well  as  how  to  best  represent  multiple  IA 
generated  plans  to  the  operator.  IMPACT’S  Spiral  2  evaluation  objectively  examines  IMPACT’S 
effect  on  operator  performance  as  compared  to  a  baseline  system. 
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2.  INTRODUCTION  &  BACKGROUND 


Future  manned  and  heterogeneous  unmanned  forces  must  be  able  to  work  increasingly  as  agile 
synchronous  teams  to  complete  missions  in  complex,  ambiguous,  and  dynamically  changing 
environments.  Advanced  and  highly  reliable  autonomous  behavior  and  multi- unmanned  vehicle 
(UxV)  cooperative  control  planning  algorithms  (CCA)  will  be  required  that  are  far  beyond  the 
capability  of  currently  fielded  systems.  Therefore,  rather  than  a  rapid  switch  from  current 
operations  to  fully  functional  autonomous  cooperative  teams,  the  likely  transition  path  will 
involve  incrementally  fielding  component  autonomous  behaviors  as  they  are  developed,  with 
overall  autonomous  capability  increasing  over  time.  Thus  a  key  challenge  is,  with  the  addition 
of  incremental  and  imperfect  autonomous  behaviors,  how  best  to  ensure  flexible,  robust  mission 
effectiveness  across  a  wide  range  of  situations  and  with  the  many  ambiguities  associated  with  the 
"fog  of  war,”  of  operational  environments. 

Mission  effectiveness  will  rely  on  increased  agility:  the  rapid  identification  and  management  of 
uncertainties  that  can  disrupt  or  degrade  an  autonomous  team’s  ability  to  safely  complete 
complex  missions.  Agility  is  especially  critical  to  robust  team  decision  making  in  highly 
challenging  and  rapidly  evolving  situations.  One  promising  method  for  increasing  agility  is  to 
establish  an  intuitive  and  effective  dialog  between  the  human  team  member  and  emerging 
autonomy.  With  this  method,  strengths  of  each  can  be  maximally  utilized  to  resolve  any 
ambiguities  and  achieve  decision  superiority,  with  autonomy  being  increasingly  unleashed  as  it 
learns  how  to  detect  and  respond  to  increasingly  ambiguous  situations. 

Many  researchers  are  exploring  critical  autonomy  components  (e.g.,  CCA,  Intelligent  Agents 
(IA)  and  human  autonomy  interfaces  (HAI))  in  isolation.  This  research,  funded  by  The  Assistant 
Secretary  of  Defense  for  Research  and  Engineering  (ASD  R&E)  under  the  Autonomy  Research 
Pilot  Initiative  (ARPI),  is  unique  in  that  it  applies  these  approaches  synergistically  to  develop  an 
Intelligent  Multi-UxV  Planner  with  Adaptive  Collaborative/Control  Technologies  (IMPACT) 
system  that  combines  flexible,  goal-oriented  “play”  calling  and  human-autonomy  interaction 
with  CCAs  that  provide  near-optimal  path  planning  solutions,  IAs  that  can  recommend  task 
assignments,  and  an  autonomies  framework  that  can  monitor  the  status  of  ongoing  tasks.  The 
IMPACT  ARPI  is  a  three  year  tri-service  (Air  Force,  Army,  and  Navy  research  laboratories) 
effort  in  researching  autonomy  with  a  specific  focus  on  agility  for  achieving  decision  superiority 
with  a  human-autonomy  team.  This  report  focuses  on  the  components  developed  and  evaluated 
in  Spiral  1  of  this  effort:  CCA,  IA,  the  Rainbow  Autonomies  Framework,  and  HAI. 

Cooperative  Control  Algorithms  (CCA) 

Recent  CCA  efforts  (Argyle,  Casbeer,  &  Beard,  2011;  Chandler,  Pachter,  &  Rasmussen,  2001; 
Frazzoli  &  Bullo,  2004;  Richards,  Bellingham,  Tillerson,  &  How,  2002;  Castanon  &  Ahner, 
2008;  Karaman,  Rasmussen,  Kingston,  &  Frazzoli,  2009;  Humphrey,  Cohen,  &  Rasmussen, 
2010;  Shima  &  Schumacher,  2009;  Krishnamoorthy,  Park,  Darbha,  Pachter,  &  Chandler,  2012) 
have  focused  on  developing  algorithms  for  multi-UxV  task  assignment  and  scheduling  that 
produce  optimal  or  near-optimal  solutions,  e.g.,  to  minimize  total  mission  time  or  total  distance 
travelled.  Typically,  such  research  focuses  on  formulating  a  joint  optimization  that  assigns 
agents  to  requested  tasks  (Shima  &  Rasmussen,  2009);  however,  game  theoretic  (Arslan, 

Marden,  &  Shamma,  2007),  rule -based  (Cortes,  Martinez,  Karatas,  &  Bullo,  2004;  Kingston, 
Beard,  &  Holt,  2008),  and  auction  (Choi,  Brunet,  How,  2009;  Kivelevitch,  Cohen,  &  Kumar, 
2011)  approaches  are  also  used.  These  capabilities  have  been  shown  to  reduce  operator 
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workload  and  the  time  required  to  generate  solutions  compared  to  performing  these  tasks 
manually.  Human  interaction  with  the  CCA,  however,  is  rather  fixed.  The  operator  identifies 
target  locations  and  how  many  vehicles  will  be  tasked  and  the  CCA  then  provides  a  plan  to  be 
accepted  or  rejected  by  the  operator.  Since  human  operators  may  have  mission  goals  such  as 
plan-to-plan  continuity  or  strict  scheduling  constraints  that  are  not  well  aligned  with  the 
underlying  CCA  optimization  metrics,  a  method  for  tuning  or  modifying  CCA  solutions  is 
needed. 

Human- Autonomy  interfaces  are  needed  that  expose  certain  mechanizations  and  parameters  of 
the  CCA  to  the  operator,  increasing  CCA  transparency  and  flexibility  over  a  large  domain  of 
missions.  Flexibility  implies  that  the  operator  has  the  ability  to  tune  or  modify  generated  play 
solutions,  expanding  beyond  the  “management  by  consent”  mode  of  interaction.  Using  the  same 
methods,  IAs  could  also  inform  the  operator  of  tuning  possibilities  and  limitations  given  the 
current  mission  environment,  and  when  circumstances  are  extremely  dynamic  or  limitations 
preclude  the  operator  from  responding  in  a  timely  manner,  the  IAs  could  react  by  modifying  the 
solution  in  a  reasonable  manner  until  additional  direction  is  provided.  This  added  flexibility  will 
lead  to  more  robust  UxV  mission  performance,  with  algorithms  that  harness  the  efficiencies 
afforded  by  the  CCA  while  providing  more  flexible  control  to  the  operator. 

In telligent  Agents  (I A) 

The  CCA  requires  that  operators  provide  surveillance  targets,  available  UxVs,  mission 
objectives,  and  desired  effects  as  initial  inputs.  The  effectiveness  of  the  CCA  is  jeopardized  by 
two  threats:  (1)  operators  may  not  know  which  plays  are  ideal  or  even  applicable  in  complex 
time-critical  situations;  and  (2)  to  avoid  re -planning,  the  CCA  must  adjust/adapt  to  local  mission 
needs  while  adhering  to  a  global  mission  solution.  Intelligent  agents  can  support  the  CCA  by 
mitigating  these  threats  and  support  operators  by  helping  them  call  plays  and  failsafe  reactions, 
while  avoiding  low-level  UxV  control. 

An  IA  can  assist  operators  by  providing  them  with  an  analysis  of  alternative  plays  (i.e.,  an 
“assistant  coach”).  Operators,  interacting  with  the  agent  through  human-autonomy  interfaces 
can  accept  suggested  plays  or  compose  their  own.  The  IA  uses  domain  knowledge  and 
inferences  about  constraints  and  possible  actions  to  assemble  sets  of  contextually  relevant  plays. 
Using  high-level  representations  of  opportunities  and  goals,  the  IA  bases  play  calling  on 
knowledge  and  reasons  operators  understand. 

Research  efforts  developing  agent-based  operator  assistance  capabilities  are  rare  (Beer,  Alboul, 
Norling,  &  Wallis,  2013).  Independent  researchers  are  investigating:  constraint-based  route 
planning  and  decision  making  (Ryan,  2010;  Fargier,  Lang,  Martin-Clouaire,  &  Schiex,  2013; 
Todorovski,  Bridewell,  &  Langley,  2012);  knowledge  tracing  (Pirolli  &  Kairam,  2013;  Sao 
Pedro,  de  Baker,  Gobert,  Montalvo,  &  Nakama,  2013);  the  use  of  abduction  during  analyses  of 
alternatives  (Menzies,  1996a;  Menzies,  1996b;  Russo,  Miller,  Nuseibeh,  &  Kramer,  2002; 
Garcez,  Russo,  Nuseibeh,  &  Kramer,  2003;  Angius,  2013);  and  the  effectiveness  of  constraint- 
based  approaches  in  intelligent  tutors  (Diziol,  Walker,  Rummel,  &  Koedinger,  2010;  Holland, 
Mitrovic,  &  Martin,  2009;  Mitrovic,  2012;  Suraweera,  Mitrovic,  &  Martin,  2010).  The  IA 
combines  features  of  these  efforts  and  advances  the  state-of-the-art  agent-based  operator 
assistance.  The  IA  is  specified  in  the  Research  Modeling  Language  (RML),  a  modeling 
language  tailored  to  the  needs  of  IA  authoring  (Douglass  &  Mittal,  2011;  Mittal  &  Douglass, 

201  la;  Mittal  &  Douglass,  201  lb).  The  IA  is  executed  in  the  Cognitively  Enhanced  Complex 
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Event  Processing  (CECEP)  modeling  and  simulation  framework  (Hansen  &  Douglass,  2012; 
Taha,  Atahary,  &  Douglass,  2013).  The  IA  makes  use  of  multiple  objective  functions  during  the 
process.  These  functions  are  used  to  develop  a  Pareto  Front  (Fundenberg  &  Tirole,  1983; 
Coello-Coello,  Lamont,  &  Veldhuizen,  2007)  that  allows  the  IA  to  provide  the  operator  with 
alternative  plays  sorted  by  estimated  costs  and  payoffs. 

Rainbow  Autonomies  Framework 

While  CCA  and  IA  can  help  select  plays,  assign  vehicles,  and  optimize  routes,  there  is  a  need  for 
automation  to  assist  in  monitoring  the  execution  of  plays.  A  modified  version  of  the  Rainbow 
Autonomies  Framework  (Lange,  Verbancsics,  Gutzwiller,  Reeder,  &  Sarles,  2012;  Lange, 
Verbancsics,  Gutzwiller,  &  Reeder,  2012)  can  support  the  operator  by  monitoring  the  status  of 
ongoing  plays.  For  example,  Rainbow  can  be  used  to  monitor  a  vehicle’s  progress  along  a  route 
against  the  plan  to  notify  the  operator  if  the  vehicle  is  falling  behind  (e.g.,  due  to  a  head  wind). 
Rainbow  has  been  enhanced  with  a  mechanism  allowing  properties  of  modeled  entities  to  be 
discovered  rather  than  programmed,  thereby  allowing  it  to  be  used  effectively  in  situations 
without  full  prior  knowledge  (Verbancsics  &  Lange,  2013). 

Human  Autonomy  Interface 

A  key  challenge  for  achieving  UxV  team  decision  superiority  is  the  design  of  a  new  interface 
paradigm,  as  current  HAI  approaches  fail  to  facilitate  trusted,  bi-directional  collaboration  and 
high-level  tasking  between  operators  and  autonomy  (Draper,  2007;  Eggers  &  Draper,  2006; 
Rodas,  Szatkowski,  &  Veronda,  2011).  For  joint  human- autonomy  teaming,  the  operator  must 
maintain  overall  situation  awareness  not  only  of  system  status  and  mission  elements  but  also  the 
intent  of  the  multiple  systems  themselves  (Chen,  Barnes,  &  Harper-Sciarini,  2011;  Endsley, 
1996).  For  situations  in  which  the  operator  needs  to  drill  into  the  autonomy,  adjust  its 
parameters,  or  override  its  operation,  intuitive  controls  (Draper,  2007)  are  required.  An  even 
more  critical  design  challenge  is  efficiently  supporting  collaborative  human-autonomy  diedog 
(Draper,  2007;  Gao,  Lee,  &  Zhang,  2006),  enabling  operators  to  monitor  and  instruct  autonomy 
in  response  to  dynamic  environments  and  goal  priorities.  An  effective  human-autonomy  dialog 
must  involve  providing  cognitive  support  and  coordination  mechanisms  with  respect  to  a  skills, 
rules,  and  knowledge  framework  (Rasmussen,  1990). 

Additionally,  the  HAI  needs  to  provide  four  support  functions  identified  by  Woods  and 
Hollnagel  (2006)  to  help  realize  an  effective  collaborative  schema.  This  includes  observability 
into  the  autonomy’s  processing  to  ensure  a  common  mental  model  and  that  both  understand  who 
is  doing  what  and  when  (Linegang  et  al.,  2006).  Directability  support  is  required  to  establish 
respective  human/autonomy  roles  in  task  completion,  with  this  allocation  dynamic  to  achieve  the 
most  effective  division  of  cognitive  functions  for  the  current  task  at  hand  (Parasuraman, 

Sheridan,  &  Wickens,  2000).  Support  in  terms  of  directing  attention  to  required  actions  is 
useful.  Finally,  support  is  required  for  the  operator’s  shifting  perspective  (Calhoun,  Warfield, 
Ruff,  Spriggs,  &  Wright,  2010;  DiVota  &  Morris,  2012)  in  multi-system  teaming.  Interfaces  that 
enable  agile  teams  to  benefit  from  maturing  CCA,  IA,  and  Rainbow  are  needed  to  provide 
mechanisms  for  the  operator  and  autonomy  to  suggest,  predict,  prioritize,  remind,  critique, 
and/or  caution  each  other  via  bi-directional  collaboration.  In  this  effort,  novel  displays  and 
controls  provide  natural,  intuitive  visibility  into  autonomy.  To  increase  autonomy ’s 
transparency  and  support  appropriate  trust  and  reliance ,  an  Ecological  Interface  Design 
(Vicente  &  Rasmussen,  1992)  approach  that  focuses  on  the  automation’s  capabilities  and 
limitations  was  used,  as  well  as  Lee’s  guidance  to  convey  the  system’s  3P’s  (purpose,  process, 
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and  performance;  Lee,  2012).  Speech  (84  words  that  could  be  combined  into  2160  potential 
phrases)  and  touch  multi-modal  controls  provide  the  operator  with  the  ability  to  flexibly  interact 
with  the  autonomy  at  any  time. 

A  key  challenge  is  designing  an  interface  that  enables  true  collaboration  for  human-autonomy 
teams.  One  potential  approach  centers  on  the  use  of  a  compiled  set  of  plans  or  “plays”  that  can 
be  modified  as  needed.  The  goal  of  this  approach  is  to  minimize  workload  associated  with 
assigning/tasking  assets,  much  like  a  football  team  has  an  approved  set  of  plays  that  facilitate 
task  delegation  by  the  quarterback  team  leader  (Miller,  Goldman,  Funk,  Wu,  and  Pate,  2004). 
The  IMPACT  interfaces  employ  this  play  concept  along  with  enabling  detailed  adaptations. 
Using  “Plays”  operators  can  initiate  a  series  of  automated  steps  or  tasks  with  a  few  verbal  and/or 
manual  commands,  e.g.,  “air  inspect  on  point  alpha”.  “ Tailoring ”  is  also  provided  whereby  the 
operator  can  inform  and/or  modify  the  cooperative  control  algorithms  and  IAs  during  play 
calling  as  well  as  revising  a  play  already  in  progress. 

IMPACT  Spiral  1 

Figure  1  showcases  how  CCA  (color  coded  yellow),  IA  (color  coded  green),  HAI  (color  coded 
blue),  and  Rainbow  (color  coded  orange)  interact  to  support  the  operator.  When  a  human 
operator  initiates  a  play  via  the  HAI,  a  play  request  is  sent  to  the  IA,  which  in  turn  queries  the 
CCA  for  the  time  it  would  take  each  UxV  that  fits  the  play  request  constraints  to  get  to  the  play 
location.  CCA  calculates  this  information  and  sends  it  back  to  the  IA,  which  then  generates  a 
solution  set.  By  default,  the  IA  will  then  request  a  route  plan  for  the  UxV(s)  it  determined  to  be 
best  to  fulfill  the  play  request.  The  CCA  generates  the  route  plan  and  it  is  then  displayed 
graphically  on  the  HAI.  The  operator  can  then  accept,  reject,  or  edit  the  play.  If  the  play  is 
accepted,  the  IA  requests  a  plan  to  execute  to  account  for  the  UxV’s  movement  while  the 
operator  deliberated.  The  CCA  updates  the  plan  and  it  is  sent  to  the  UxV  to  execute.  Upon  play 
execution,  Rainbow  monitors  the  UxVs  progress  against  the  plan. 


DISTRIBUTION  STATEMENT  A.  Approved  for  public  release. 


5 


Cleared.  88PA,  Case  #2017-1 156. 


Operator 
calls  a  play 
via  HAI 


Sends  request/constraints 
and  requests  solution  set 


CCA 


IA 


HAI 


Rainbow 


IA 


IA  generates 
solution  set 


Rejects/Edits  Play 


Requests  time  to  get  to  play 
location  for  all  UxVs  that 
fit  play  request  constraints 

CCA 

Provides  time  to  get  to  play 
locations  for  all  UxVs  that  fit 
play  request  constraints 

Requests  route  plan(s) 
for  best  vehicle(s) 


HAI  graphically 

represents  play 

◄ - 

in  sandbox 

V 

CCA 


Provides  route  plan(s)  for 
best  vehicle(s) 


Accepts  Play  „ 

IA  requests 
plan  to  execute 

CCA  updates 
plan  to  accountfor 
vehicle  movement 

3 

r 

Rainbow 
monitors 
ongoing  play 

HAI  sends  plan  to 
UxV(s)  to  execute 

Figure  1.  How  IMPACT  components  interact  in  response  to  a  play  call. 

To  put  this  interaction  among  components  in  context,  Figure  2  represents  an  operator  responsible 
for  controlling  seven  UxVs,  tasked  with  providing  surveillance  at  Point  Charlie  at  night  within 
15  minutes  (2a).  The  IA  first  determines  which  vehicles  have  the  necessary  capabilities  (in  this 
case  an  infrared  (IR)  sensor)  to  perfonn  the  task  and  eliminates  those  which  don’t  (FN-18  and 
FN-20)  from  consideration  (2b).  The  IA  then  sends  a  request  to  CCA  to  determine  how  long  it 
would  take  each  of  the  capable  vehicles  to  reach  Point  Charlie  and  eliminates  the  vehicles  that 
can’t  reach  Point  Charlie  in  time  (FN-13  and  FN-15)  from  consideration  (2c).  Next,  the  IA 
considers  how  long  each  vehicle  can  remain  at  Point  Charlie  based  on  fuel  bum  rate  and  fuel 
level  and  selects  the  vehicle  (FN-12)  that  can  remain  the  longest  (2d).  The  IA  then  sends  a 
request  to  CCA  to  plan  a  route  for  FN-12  to  reach  Point  Charlie  (2e),  which  the  operator  accepts. 
Finally,  as  FN-12  travels  to  Point  Charlie  the  Rainbow  Autonomies  framework  monitors  the 
situation  and  notices  a  popup  restricted  operating  zone  (ROZ)  that  requires  FN-12  to  change  its 
route  (2f). 
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Figure  2.  An  example  of  how  IMPACT  components  recommend  a  vehicle,  plan  a  route,  and 
monitor  task  status. 

The  present  evaluation  was  designed  to  solicit  feedback  from  UxV  operators  and  base  defense 
subject  matter  experts  on  the  Spiral  1  IMPACT  system  that  integrates  human-machine  interfaces, 
CCA,  IA,  and  the  Rainbow  autonomies  framework  in  the  context  of  a  base  defense  mission. 

This  feedback  will  be  used  to  inform  the  development  of  the  Spiral  2  IMPACT  system. 
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3.  METHODS,  ASSUMPTIONS  AND  PROCEDURES 

Participants 

Seven  volunteers  from  Wright-Patterson  Air  Force  Base  (WPAFB)  participated  in  the  study. 
Three  participants  had  prior  experience  flying  UAVs  (Predator,  ScanEagle,  Global  Hawk,  and 
Shadow)  as  well  as  manned  aircraft  (F-l  1 1,  C-130,  T-38,  T-l,  and  RF-4).  Four  participants  were 
active  Air  Force  security  force  personnel  with  experience  in  conducting  base  defense  operations 
in  deployed  environments  (Afghanistan,  Germany,  Iraq,  Kuwait,  and  Saudi  Arabia).  Relevant 
for  this  evaluation,  one  UAV  operator  had  experience  using  a  ScanEagle  to  support  base  defense 
missions.  All  participants  were  male  and  reported  normal  or  corrected-to-normal  vision,  normal 
color  vision,  and  normal  hearing.  Test  procedures  were  approved  by  the  Air  Force  Institutional 
Review  Board  and  all  participants  provided  informed  consent  prior  to  participation. 

Equipment 

The  IMPACT  test  bed  used  the  Fusion  software  developed  by  AFRL  71 1HPW/RHCI  (for  an  in- 
depth  description  of  Fusion,  see  Rowe,  Spriggs,  &  Hooper,  2015).  The  IMPACT  test  bed  used 
six  computers  (Single  Dell  T5610  &  4  Dell  R7610’s  running  Windows  8.1).  One  computer  ran 
the  HAI,  the  IA,  the  CCA,  the  Rainbow  Autonomies  framework,  and  the  AMASE  [AVTAS 
(Aerospace  Vehicle  Technology  Assessment  and  Simulation)  Multi-Agent  Simulation 
Environment]  vehicle  simulation  (used  to  simulate  the  UxVs).  One  computer  ran  the  test 
operator  console  and  simulation  for  simulated  entities  in  the  sensor  videos  (Vigilant  Spirit 
Simulation;  Feitshans  &  Davis,  2011),  three  computers  ran  two  simulated  sensor  videos  each 
(SubrScene  Image  Generator;  www.subrscene.org/),  and  one  computer  ran  an  XMPP  (Extensible 
Messaging  and  Presence  Protocol)  Chat  server  for  simulated  communications.  The  IMPACT 
test  bed  interface  used  four  27”  touchscreen  monitors  (Acer  T272HUL),  a  headset  with  a  boom 
microphone  (Plantronics  GameCom  Commander),  a  foot-pedal  (for  push-to-talk  speech  control), 
and  a  mouse  and  keyboard. 

An  overview  of  the  IMPACT  test  bed  interface  is  shown  in  Figure  3.  Starting  with  the  top  screen 
and  moving  clockwise,  the  Tactical  Situation  Awareness  Display  provided  a  geo-referenced  map 
with  UxV  locations  as  well  as  UxV-specific  information  (e.g.,  a  UxV’s  current  play,  error 
indicators,  a  UxV’s  planned  route,  etc.).  The  Payload  Management  display  showed  available 
sensor  feeds  on  demand  and  will  include  other  payloads  (e.g.,  weapons,  communication  devices, 
etc.)  in  the  future.  The  Sandbox  display  was  a  workspace  for  the  operator  to  call  and  edit  plays 
without  obscuring  the  current  state  of  the  world  (which  was  always  available  in  the  Tactical 
Situation  Awareness  Display).  Finally,  the  System  Tools  display  contained  chat  windows  as 
well  as  help  documentation  (e.g.,  list  of  voice  commands). 
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Figure  3.  A  screenshot  of  IMPACT’S  test  bed  interface,  with  the  tactical  situation 
awareness  display  on  the  top  monitor,  the  payload  management  display  on  the  right 
monitor,  the  sandbox  (planning  area)  on  the  bottom  monitor,  and  system  tools  on  the  left 
monitor. 


IMPACT  Spiral  1  Human-Autonomy  Interface 

The  ensuing  section  provides  a  high-level  description  of  the  IMPACT  Spiral  1  play  calling 
interface  (see  Calhoun,  Ruff,  Behymer  &  Frost,  in  press  for  a  description  of  the  Spiral  2 
interface).  The  IMPACT  operator  was  responsible  for  six  UxVs  during  the  Spiral  1  evaluation 
(see  Figure  4):  three  UAVs  (call  signs  FN-41,  FN-42,  and  FN-43;  represented  by  the  diamond 
shaped  icons),  two  UGVs  (call  signs  TR-11  and  TR-21;  represented  by  the  tombstone  shaped 
icons)  and  one  USV  (call  sign  SY-41;  represented  by  the  pentagon  shaped  icon). 


Figure  4.  UAY,  UGY,  and  USV  icons  displayed  on  the  tactical  situation  awareness  display. 
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In  Spiral  1,  operators  initiated  plays  by  clicking  or  touching  one  of  thirteen  icons  in  the  Play 
Creator  interface  (see  Figure  5a),  each  of  which  corresponded  to  a  specific  play.  For  example,  to 
call  an  air  surveillance  at  a  point,  operators  clicked  or  touched  the  leftmost  icon  in  the  first  row. 
Operators  could  also  initiate  plays  by  using  the  voice  command  (e.g.,  “air  surveillance”) 
associated  with  each  play.  Once  a  play  was  initiated,  operators  selected  a  play  location  from  a 
dropdown  menu  of  previously  identified  points  or  created  a  new  point  by  clicking  on  the  map 
(see  Figure  5b). 


Figure  5.  Play  Creator  Tile,  (a)  icons  for  each  of  Spiral  l’s  thirteen  plays,  (b)  methods  for 
identifying  a  location  including  selecting  any  point  on  the  map  or  selecting  an  existing 
location  via  a  dropdown  menu. 


Once  the  operator  selected  a  point  (e.g.,  Point  Alpha),  the  IA  sent  a  request  to  CCA  asking  the 
time  it  would  take  each  UAV  to  reach  Point  Alpha,  and  by  default  recommended  the  UAV  that 
would  arrive  soonest,  in  this  case  FN-42  (see  Figure  6). 


Priority  set  to  normal 


Figure  6.  Play  Creator  Intelligent  Agent  Reasoning  —  the  operator  has  asked  the 
Intelligent  Agent  to  optimize  for  time  (the  stop  watch  icon  is  selected)  and  has  set  priority 
to  normal,  which  allowed  the  Intelligent  Agent  to  recommend  the  optimal  vehicle  based  on 
these  presets. 
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Additionally,  the  CCA  planned  route  from  FN-42’s  current  location  to  Point  Alpha  was 
represented  on  the  map  with  a  dashed  line  (see  Figure  7).  Each  potential  play  was  given  a 
unique  play  color  (in  this  case  orange). 


Figure  7.  Planned  route  for  FN-42  to  Point  Alpha  (represented  by  a  dashed  orange  line). 

The  operator  had  the  option  to  override  the  IA’s  recommendation  by  manually  selecting  a 
vehicle  using  the  Play  Creator’s  vehicle  panel.  In  Figure  8,  the  operator  manually  selected  FN- 
41  to  complete  the  task  and  CCA  immediately  planned  a  route  from  FN-41’s  current  location  to 
Point  Alpha,  again  represented  by  the  dashed  orange  line  (see  Figure  9). 


Figure  8.  Play  Creator  Tile  manual  vehicle  selection  —  the  operator  has  opened  the  manual 
vehicles  panel  and  selected  FN-41. 
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Figure  9.  Planned  route  for  FN-41  to  Point  Alpha  (represented  by  a  dashed  orange  line). 

The  operator  also  had  the  ability  to  improve  the  IA’s  recommendation  by  providing  updated 
situation  information.  For  example,  if  FN-41  ’s  Sensor  Operator  reported  heavy  cloud  coverage 
at  Point  Alpha,  the  IMPACT  operator  communicated  this  to  the  IA  by  selecting  the  cloud  icon  in 
the  Play  Creator  and  the  IA  now  recommended  FN-43,  which  has  an  IR  sensor  (see  Figure  10) 
and  CCA  planed  a  route  for  FN-43  to  get  to  Point  Alpha  (see  Figure  1 1). 


Play  Creator 


Air  Surveillance 


Intelligent  Agent  updates 
recommendation  to  FN-43 
which  has  an  IR  sensor 


©“ 

Environment: 

Optimization: 

Play  Priority: 


IPlay  Priority: 


Operator  informs 
the  Intelligent  Agent  of 
environmental  conditions 


Figure  10.  Operator  providing  information  to  the  Intelligent  Agent  including  enviromental 
conditions  (the  cloud  icon  is  selected),  optimization  preference  (the  stopwatch  icon  is 
selected),  and  prioirty  (normal  is  sleeted).  The  Intelligent  Agent  has  recommened  FN-43 
(which  has  an  IR  sensor)  based  on  these  operator  inputs. 
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Figure  11.  Planned  route  for  FN-43  to  Point  Alpha  (represented  by  a  dashed  orange  line). 

The  operator  could  also  edit  route  details  manually  using  the  Play  Creator’s  details  panel  (see 
Figure  12).  In  this  example  the  operator  has  changed  the  loiter  length  to  1500  meters,  creating  a 
racetrack  loiter  pattern  (see  Figure  13). 


Figure  12.  Play  Creator  details  panel  —  the  operator  has  opened  up  the  details  panel  and 
changed  the  loiter  length  to  1500  m. 
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Figure  13.  Racetrack  loiter  pattern  (represented  by  the  dashed  orange  line). 

Once  modifications  were  completed,  the  operator  could  click  the  checkmark  to  accept  the  plan 
and  FN-43  would  begin  to  execute  it,  with  the  change  from  plan  to  reality  represented  by  the 
dashed  line  becoming  solid  and  FN-43’ s  icon  changing  to  orange  (see  Figure  14). 


Figure  14.  Plan  Execution  (the  dashed  orange  line  has  changed  to  solid  and  the  vehicle 
color  has  changed  to  orange  now  that  the  play  is  executing). 

Additionally,  an  Active  Play  Manager  (see  Figure  15)  opens  and  lists  the  plays  that  are  currently 
executing.  Once  a  play  is  active  the  Rainbow  Autonomies  framework  monitors  the  play’s 
progression  and  checks  to  see  if  FN-43 ’s  estimated  time  enroute  (ETE),  fuel  state,  and  dwell 
time  (how  long  FN-43  can  remain  at  Point  Alpha)  are  going  as  planned. 
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Figure  15.  The  Active  Play  Manager  provides  a  list  of  all  ongoing  plays. 

Clicking  a  play  within  the  Active  Play  Manger  opens  a  window  that  provides  status  information 
about  the  play  (Figure  16;  see  Behymer,  Ruff,  Mersch,  Calhoun,  &  Spriggs,  2015  for  additional 
details).  This  chart  conveys  several  pieces  of  information  to  the  operator.  In  the  bar  chart,  the 
parameters  were  ordered  from  left  to  right  based  on  their  priority,  with  the  highest  priority 
parameter  at  the  far  left.  The  width  of  each  column  also  represented  the  priority  of  each 
parameter.  For  example,  estimated  time  enroute  (ETE)  was  given  the  most  weighting,  followed 
by  fuel  and  dwell  time.  Color  is  used  to  represent  a  play’s  status.  If  everything  is  going 
according  to  plan  (e.g.,  the  UV  is  still  expected  to  arrive  at  task  location  on  time)  the  middle 
segment  of  that  parameter’s  column  is  green  and  indicates  a  “normal/ideal”  state.  A  warning 
state  is  represented  with  three  yellow  segments  and  indicates  a  “slight”  deviation  from  the  ideal 
state.  An  error  state  is  represented  with  five  red  segments  and  indicates  a  “severe”  deviation 
from  the  ideal  state.  The  specific  location  of  the  segment  with  the  brighter,  more  saturated  color 
indicates  whether  the  value  exceeds  or  is  less  than  the  desired  operating  range  for  that  parameter. 


Figure  16.  Clicking  a  play  in  the  Active  Play  Manager  opens  up  an  additional  tile 
containing  detailed  play  and  vehicle  status  information. 


Procedure 

Participants  completed  a  background  questionnaire  (see  Appendix  A)  and  were  given  an 
overview  of  the  IMPACT  test  bed.  The  overview  included  a  slideshow  that  described  the 
project’s  goals  and  introduced  the  concept  of  play  calling.  Next,  participants  were  seated  at  the 
IMPACT  test  bed  and  given  an  overview  of  their  mission  that  included: 
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•  A  description  of  the  UxVs  they  would  be  controlling,  how  each  UxV,  its  route,  and  its  sensor 
footprint  were  represented  on  the  map,  and  the  tasks  that  each  UxV  was  responsible  for 
performing  in  support  of  base  defense  operations. 

•  An  overview  of  the  base  they  would  be  defending  including  the  base’s  perimeter,  sectors, 
critical  facilities,  patrol  zones,  and  the  named  areas  of  interests  in  the  area  immediately 
surrounding  the  base. 

•  An  overview  of  their  role  as  a  multi-UxV  operator  supporting  base  defense  operations  that 
described  that  they  would  be  assigning  high-level  tasks  to  the  UxVs  while  the  autonomous 
system  components  flew,  drove,  and  steered  the  UxVs.  Also,  that  they  would  be  assigned 
tasks  from  their  commander  in  a  chat  window  and  that  they  would  have  access  to  the  UxV 
sensor  feeds  but  it  was  not  their  responsibility  to  monitor  them. 

The  experimenter  then  provided  the  participant  with  a  high-level  description  of  the  IMPACT  test 
bed’s  layout  (similar  to  the  above  description  of  Figure  3).  Next,  the  experimenter  explained  the 
following  IMPACT  components  while  the  participant  sat  at  the  controls  performing  each 
function. 

•  Chat  Window,  the  location  of  the  chat  window,  how  to  send  and  receive  chat  messages, 
audio  alerts  associated  with  incoming  chat  messages,  how  to  resize  the  chat  window,  how 
to  change  the  font  size  of  the  chat  window’s  text. 

•  Play  Creation  Tile  (described  above  in  Figure  5a):  how  to  call  plays,  how  play  types  were 
categorized  by  row  (e.g.,  the  first  row  contains  plays  focused  on  points,  the  second  on 
routes,  the  third  on  areas,  the  fourth  on  sensors,  and  the  fifth  on  custom  plays  like  normal 
full  coverage  and  highly  mobile),  how  hovering  over  a  play  icon  will  provide  the  play’s 
name,  how  to  specify  a  play  location,  specific  vehicles,  constraints,  and  details. 

•  Play  Icon  Symbology:  how  the  play’s  type  is  represented  by  a  shape  within  the  play  icon 
(e.g.,  plus  sign  for  point  plays,  lines  for  route  plays,  squares  for  area  plays,  and  a  sensor 
footprint  for  sensor  plays),  how  the  unmanned  vehicle  type  associated  with  a  play  is 
represented  by  a  shape/location  (UAVs  -  upper  left/diamond,  UGVs,  lower 
left/horizontal  rectangle,  USV  -  lower  right/vertical  rectangle). 

•  Play  Calling:  how  plays  could  be  called  by  clicking/touching  a  play  icon  in  the  play 
creation  tile  or  by  using  a  voice  command.  Participants  were  trained  on  how  to  activate 
the  speech  recognition  system  (pushing  a  foot  pedal  or  clicking/touching  a  bar  on  the 
bottom  of  the  sandbox)  as  well  as  how  the  speech  recognition  system  provided  feedback 
(verbally  over  the  headset  as  well  as  text  on  the  screen). 

•  Play  Tile:  how  information  about  a  play’s  progress  could  be  obtained,  how  plays  could 
be  edited. 

•  IA  Recommendations:  how  the  IA  would  recommend  a  play  if  a  UxV  was  near  a  critical 
facility. 

Next,  participants  were  trained  on  calling  plays  using  each  modality  (mouse  and  keyboard, 
touch,  and  voice)  in  order  to  familiarize  them  with  each  method  of  play  calling.  Participants 
received  12  chat  messages  (e.g.,  “Using  voice,  call  an  air  surveillance  at  Point  Alpha”)  asking 
them  to  call  a  play  using  a  specific  modality  (4  plays  for  each  modality).  A  list  of  voice 
commands  was  available  to  the  participants  on  the  left  monitor.  Table  1  lists  the  exact  sequence 
of  plays  participants  were  asked  to  call  during  this  portion  of  the  training  as  well  as  the  modality 
participants  were  instructed  to  use  for  each  of  the  12  plays. 
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_ Table  1.  Play  Calling  Modality  Training _ 

Method  Play 

Voice  Air  Surveillance  at  Point  Alpha 

Voice  Normal  Full  Coverage  Patrol 

Voice  Ground  Inspect  at  Route  Bravo 

Voice  Air  Expanding  Square  Search  at  Point  Charlie 
Touch  Air  Sector  Search  at  Point  Alpha 

Touch  Surface  Parallel  Search  at  Area  Delta 

Touch  Air  Parallel  Search  at  Area  Bravo 

Touch  Ground  Inspect  at  Point  Charlie 

Mouse  Air  Ground  Surveillance  at  Point  Bravo 
Mouse  Air  Inspect  at  Route  Charlie 

Mouse  Air  Surface  Parallel  Search  at  Area  Delta 
Mouse  Go  Highly  Mobile  at  Point  Alpha 

Participants  were  then  trained  on  how  to  specify  constraints,  vehicles,  and  details  when  calling 
and/or  editing  a  play.  For  each  modality,  participants  were  asked  via  chat  message  to  call  a 
specific  play  (e.g.,  “Using  voice,  call  an  air  surveillance  on  Point  Alpha,  set  sensor  to  EO,  and 
optimize  for  low  impact”),  then  make  edits  to  the  ongoing  play  (e.g.,  “Change  the  loiter  type  to  a 
figure  8”).  The  exact  sequence  of  the  play  calls  and  play  edits  participants  were  asked  to  make  is 
listed  in  Table  2.  If  a  participant  made  a  mistake,  the  experimenter  provided  feedback  and  the 
participant  tried  again  until  he  had  successfully  completed  the  correct  action.  On  average, 
training  lasted  1  hour.  Before  continuing,  participants  took  a  short  break  that  was  also  used  to 
reset  the  test  bed  for  the  mission  simulation. 

_ Table  2.  Play  Editing  Training _ 

Play  Call  or  Edit 

Air  Surveillance  on  Point  Alpha  with  EO  that  minimizes  impact  to  other  plays 
Edit-  Change  loiter  type  to  figure  8 
Edit-  Change  standoff  distance  to  2000  m 
Edit-  Change  visibility  to  obscured 

Edit-  Change  loiter  type  to  race  track,  change  standoff  distance  to  0  m, 
change  visibility  to  clear,  and  change  loiter  length  to  1000  m 

Mouse  Expanding  Square  Search  at  point  bravo,  size  5,  cloudy,  minimize  fuel 
Any  Edit-  Change  extent  to  1500  m 
Any  Edit-  Change  vehicle  to  a  different  UAV 
Any  Edit-  Change  extent  to  2000  m  and  select  SAR  sensor 

Touch  Air  Parallel  Search  at  Area  Delta 
Any  Edit-  Change  the  air  vehicle  to  a  different  UAV 

Any  Edit-  Change  the  sweep  angle  to  30  degrees 

Any  Edit-  Change  the  sensor  to  IR  and  the  visibility  to  obscured 

Mission 

The  purpose  of  the  mission  was  to  exercise  IMPACT  capabilities  within  a  realistic  context.  To 
achieve  a  realistic  simulation,  the  mission  scenario  was  developed  based  on  the  results  of  a 
cognitive  task  analysis  (CTA)  conducted  prior  to  the  study.  Three  CTA  methods  were  used  to 


Method 

Voice 

Any 

Any 

Any 

Any 
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gain  an  understanding  of  current  base  defense  operations  as  well  as  how  operators  might 
integrate  UxVs  into  their  operations.  First,  the  task  diagram  method  (Militello,  Hutton,  Pliske, 
Knight,  Klein,  &  Randel,  1997)  was  used  to  elicit  the  major  job  components  of  base  defense 
personnel  and  identify  the  major  cognitive  challenges  base  defense  personnel  face.  Next,  the 
critical  decision  method  (Klein,  Calderwood,  &  MacGregor,  1989)  was  used  to  identify  critical 
incidents  that  security  forces  had  experienced  in  order  to  inform  the  development  of  realistic 
scenarios  for  the  evaluation.  Finally,  the  simulation  interview  method  (Militello,  Hutton,  Pliske, 
Knight,  Klein,  &  Randel,  1997)  was  used  to  investigate  how  security  force  personnel  would  use 
UxVs  in  the  context  of  a  notional  base  defense  scenario.  Using  PowerPoint  slides,  experimenters 
provided  interviewees  with  a  mock  base  and  asked  them  how  they  would  defend  it  using  current 
methods  (i.e.,  without  UxVs).  Experimenters  then  briefed  participants  on  the  capabilities  of  a 
UAV  (Boeing’s  ScanEagle),  an  unmanned  ground  vehicle  (General  Dynamics  Mobile  Detection 
Assessment  and  Response  System  [MDARS]),  and  an  unmanned  surface  vehicle  (BAE  System’s 
Protector)  and  asked  how  they  would  incorporate  these  vehicles  into  the  base  defense  operations 
for  this  notional  base.  Interviewees  were  then  given  a  series  of  simulated  events  (e.g.,  a  crowd  is 
forming  outside  a  base  gate,  a  HUMINT  (human  intelligence)  report  of  a  military  age  male 
carrying  a  mortar  within  10  miles  of  the  base)  and  asked  how  they  would  respond  to  the  events 
using  UxV  assets. 

Prior  to  the  mission,  participants  were  given  a  pre-mission  brief  that  informed  them  that  their 
tasks  would  be  to  call  plays  in  response  to  chat  messages  from  their  remotely  located  commander 
and  sensor  operator,  played  by  confederates.  Participants  were  also  informed  that  the  scenario 
would  begin  with  a  UAV  investigating  a  suspicious  watercraft  and  all  other  UxVs  on  high  alert 
(i.e.,  a  highly  mobile  patrol  pattern).  Table  3  lists  the  exact  sequence  of  mission  events  that 
occurred.  Participants  were  instructed  to  respond  to  chat  messages  as  soon  as  possible  and  were 
free  to  respond  to  commander  and  sensor  operator  requests  with  the  play  (called  with  the 
modality  of  their  choice)  they  felt  best  achieved  the  commander/sensor  operator’s  goal.  The 
mission  lasted  approximately  20  minutes. 

_ Table  3.  Sequence  of  Mission  Events _ 

Mission  Events _ 

1.  Operator  receives  a  chat  message  from  the  Sensor  Operator  that  the 
unidentified  watercraft  is  a  fishing  boat. 

2.  Operator  receives  a  chat  message  from  the  Commander  to  resume 
normal  base  defense  operations/patrols. 

3.  The  IA  recognizes  a  serendipitous  surveillance  opportunity  (a  UAV  is 
near  a  critical  facility)  and  recommends  a  play  (air  surveillance  at  the 
critical  facility)  to  the  operator. 

4.  Operator  receives  a  chat  message  from  the  Commander  to  send  a 
UAV  to  Point  Charlie  and  to  instruct  other  UxVs  to  go  highly  mobile 
in  response  to  a  patrol  reporting  smoke  at  Point  Charlie. 

5.  Operator  receives  a  chat  message  from  the  Sensor  Operator  to  send  a 
UGV  to  Point  Charlie  to  get  a  better  look. 

6.  Operator  receives  a  chat  message  from  the  Commander  to  provide  the 
UGV  headed  to  Point  Charlie  with  a  UAV  overwatch. 

Once  the  mission  was  finished,  participants  completed  paper  questionnaires  on  the  overall 
IMPACT  system  (see  Appendix  B)  and  its  components  (see  Appendix  C).  Then,  a  semi- 
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structured  interview  (see  Appendix  D)  was  conducted  to:  capture  additional  feedback  on 
IMPACT  and  the  associated  technologies,  solicit  feedback  on  the  scenario  and  base  defense 
domain,  and  debrief  participants.  Finally,  participants  were  briefed  on  additional  IMPACT 
interface  features  and  concepts  that  were  under  development  but  not  yet  integrated.  Participants 
were  asked  to  provide  input/feedback  on  the  components’  designs  for  refinement  prior  to 
integration  (input/feedback  on  components  not  integrated  into  Spiral  1  are  outside  the  scope  of 
this  report).  The  questionnaires,  interview/debrief,  and  slideshow  presentations  lasted  100 
minutes  on  average.  The  entire  procedure  lasted  approximately  3.25  hors  overall. 
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4.  RESULTS 

Of  the  seven  individuals  who  participated  in  the  study,  six  completed  the  training  and  mission  in 
the  allotted  time.  Due  to  unanticipated  time  restrictions,  the  seventh  participant  was  unable  to 
complete  the  study  and  was  eliminated  from  the  data  analysis.  Due  to  the  small  number  of 
participants,  UV  operators  and  Security  Force  personnel  data  were  not  analyzed  separately.  In 
general,  the  UV  operators  had  more  suggestions  on  improving  IMPACT’S  HAI  and  autonomous 
components  while  Security  Force  personnel  had  more  suggestions  on  improving  the  mission 
scenario. 


Overall  System 

Participants  used  a  5-point  Likert  scale  (ranging  from  1:  No  Aid  to  5:  Great  Aid)  to  rate 
IMPACT’S  potential  value  for  future  UxV  operations  as  well  as  the  ability  of  IMPACT  to  aid 
operator  workload  and  situation  awareness  (see  Figure  17;  note  that  the  vertical  line  is  the  scale’s 
midpoint,  so  all  ratings  to  the  right  are  above  a  3).  Participant  responses  indicated  that  they  had  a 
positive  opinion  on  the  potential  value  of  IMPACT  for  future  UxV  operations,  to  aid  workload, 
and  to  aid  situation  awareness,  with  no  ratings  less  than  a  4. 


«  No  Aid 


Great  Aid  » 


5:  Great 
Aid 


□  3 

□  2 


□ 


1:  No 
Aid 


Figure  17.  Participant  ratings  for  IMPACT’S  ability  to  aid  situation  awareness,  workload, 
and  potential  value  to  future  UxV  operations.  Every  participant  rated  IMPACT  at  a  4  or 
higher  for  each  of  the  three  measures.  The  numbers  inside  the  bars  indicate  the  number  of 
participants  who  provided  a  rating  at  that  scale  value. 

The  overall  usability  of  IMPACT  was  assessed  using  the  System  Usability  Scale  (SUS;  Brooke, 
1996).  The  SUS  asks  participants  to  evaluate  10  items  related  to  system  usability  using  a  5  point 
Likert  scale  ranging  from  Strongly  Agree  to  Strongly  Disagree  (see  Figure  18),  and  these  10 
items  contribute  to  an  overall  SUS  score.  The  overall  mean  SUS  score  for  IMPACT  was  73.75, 
placing  it  in  the  70th  percentile  of  SUS  scores. 
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Strongly 

Agree 


Strongly 

Disagree 


Like  to  Use 
the  System 
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Complex 


Easy  to  Use  Do  not  need  Well 

a  Support  Integrated 
Person 


Consistent  Easy  to  Learn  Not  Confident  Didn't  Need 

Cumbersome  Using  the  to  Learn  a 
to  Use  System  Lot  to  Get 
Going 


Figure  18.  System  Usability  Scale  Results.  (Error  Bars  are  the  Standard  Errors  of  the 
Means) 


Participants  were  also  asked  what  they  most  liked,  what  they  least  liked,  what  was  most 
confusing,  and  what  they  would  improve  in  regards  to  the  overall  IMPACT  system  (see  Table  4). 


Tab! 


le  4.  Participant  Comments  on  the  Overall  System 


Most  Liked 

Least  Liked 

Most  Confusing 

Improvements 

•  Multiple  modalities  (e.g., 
speech,  touch,  keyboard 
and  mouse) 

•  Agent  recommending 
plays  when  a  UV  is  near 
a  critical  facility 

•  Play  Creation  tile’s 
intuitive  symbology 

•  Ability  to  maintain  big 
picture  on  top  screen 
while  zooming  in  and 
planning  on  bottom 
screen  (sandbox) 

•  Touch  screen 
wasn’t  precise 
enough;  difficult  to 
select  correct 
button 

•  Cannot  manually 
draw  routes  for 
UxVs 

•  Displaying  all  UxV 
routes  made  the 
map  cluttered 

•  Requiring 
confirmation  to 
execute  a  play 

•  Difficult  to  determine 
where  a  specific  UxV 
was  going  due  to  map 
clutter  (e.g.,  vehicle 
routes,  vehicle  loiter 
patterns) 

•  Challenge  to  learn 
how  play  icons  were 
organized  in  Play 
Creator  tile 

•  Ability  to  call  plays 
by  clicking 
locations/vehicles  on 
the  map 

•  A  single  ear  headset 
would  be  more 
comfortable  and  help 
maintain  situation 

awareness 

•  Expand  voice 
commands  to  more 
than  just  play  calling 

•  Forecasting 
capabilities  (e.g.,  what 
are  things  going  to  be 
like  in  10  minutes) 

In  addition  to  rating  the  overall  IMPACT  system,  participants  were  asked  to  rate  four  system 
components  (Play  Calling,  Autonomy,  Feedback,  and  Test  Bed)  on  five  parameters  (Potential 
Value,  Ease  of  Use,  Integration,  Consistency,  and  Ease  of  Learning)  and  provide  any  comments 
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they  had  about  each  component.  The  results  for  each  system  component  are  described  below, 
accompanied  by  a  high-level  summary  of  participant  comments.  Overall,  88%  of  ratings  were 
either  a  4  or  5  (the  top  two  categories)  and  only  a  single  component  (Ease  of  Learning  of 
Feedback)  was  rated  less  than  a  3. 

Play  Calling 

Participants  were  asked  to  rate  the  HAI  components  that  allowed  them  to  issue  high  level  plays 
to  one  or  more  UxVs  including  the  play  creator  tile  (Figure  5a),  play  creation  work-flow  (Figure 
6),  preset  panel  (Figure  10),  selecting  a  play  location  (Figure  5b),  selecting  a  play  vehicle  (Figure 
8),  modifying  play  details  (Figure  12),  and  confirming  the  play  (Figure  14).  Participants  rated  all 
five  parameters  positively  (see  Figure  19). 


Consistent 


5 


Well  Integrated 


3  2 


5:  Strongly 
Agree 


□  3 

□  1:  Strongly 
Disagree. 


Figure  19.  Participant  Ratings  for  Play  Calling  —  90%  of  participant  ratings  across  the 
five  measures  were  4  or  higher.  The  numbers  inside  the  bars  indicate  the  number  of 
participants  who  provided  a  rating  at  that  scale  value. 

In  general  participants  responded  positively  to  the  Play 
Creator  tile  icon  symbology  (Figures  5a  and  6), 
commenting  that  the  symbols  were  more  intuitive  and  easier 
to  learn  than  a  text-based  menu  system.  In  fact,  one 
participant  even  suggested  replacing  the  text  fields  in  the 
details  panel  (Figure  12)  with  a  symbology  based  approach. 

Participants  made  several  recommendations  for 
improvements  including  organizing  play  icons  by  vehicle 
type  (e.g.,  all  UAV  plays  in  the  first  row,  UGV  plays  in  the  second  row,  etc.)  instead  of  by  play 
type  (e.g.,  all  point  plays  in  the  first  row,  route  plays  in  the  second  row,  etc.),  adding  a  visual 
indicator  when  a  play  detail  has  been  modified  from  the  default  setting,  and  adding  a  warning 
before  closing  a  play  during  editing  if  the  changes  have  not  been  executed  (i.e.,  sent  to  the  UxV). 


“7  do  like  the  heavy  reliance 
on  symbology.  The 
learning  cur\>e  is  quicker. 

Less  numbers  and  words  is 
good."  -  UAV  Operator 
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Autonomy 

Participants  were  asked  to  rate  the  IMPACT’S  autonomous  components,  including  the  automated 
route  planning  (CCA),  vehicle  and  play  recommendation  features  (IA),  and  play  monitoring 
(Rainbow).  Participants  generally  rated  all  five  parameters  positively  (see  Figure  20). 


5:  Strongly 
Agree 


□  3 

□  2 

□  1:  Strongly 
Disagree 


«  Disagree 


Agree  » 


Figure  20.  Participant  Ratings  for  IMPACT’S  Autonomous  Capabilties  —  93%  of 
participant  ratings  across  the  five  measures  were  4  or  higher.  The  numbers  inside  the  bars 
indicate  the  number  of  participants  who  provided  a  rating  at  that  scale  value. 
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I  find  myself  falling  into,  and  it  may  be  a 
trap  or  it  may  be  not,  of  trusting  the  system. 
Partly  I  was  inherently  believing  that  it  was 
going  to  be  a  good  solution,  and  it  wasn  7 
easy  to  double  check  what  was  going  on."  - 
UAV  Operator 


Overall,  participants  commented  favorably  on 
IMPACT’S  autonomous  capabilities,  citing  IA  vehicle 
and  play  recommendations  as  particularly  valuable. 

However,  participant  comments  reflected  that  they 
often  felt  disconnected  from  the  IA’s  reasoning 
process.  For  example,  if  the  operator  selected  both  the 
cloudy  optimization  preset  and  the  EO  sensor  preset 
the  IA  wouldn’t  provide  a  recommendation  because  it 
was  impossible  to  generate  a  plan  that  could  fulfill 

both  these  constraints.  Unfortunately,  the  operator  only  received  a  message  that  the  play  request 
was  over  constrained,  not  why  the  request  couldn’t  be  fulfilled.  Additionally,  several 
participants  were  unsure  how  they  could  evaluate  the  IA’s  recommendations  as  they  received  no 
feedback  on  why  the  IA  had  recommended  one  vehicle  over  another  to  fulfill  a  play  request. 
Finally,  some  participants  suggested  that  providing  multiple  courses  of  action  to  choose  from 
may  be  preferable  to  providing  a  single  course  of  action  in  some  situations. 
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Feedback 


Participants  were  asked  to  rate  IMPACT’S  ability  to  provide  feedback  on  play  status  via  the 
Active  Play  Manager  (Figure  15)  and  the  Play  Status  Tile  (Figure  16).  Participants  generally 
rated  all  five  parameters  positively  (see  Figure  21). 


«  Disagree  Agree  » 


5:  Strongly 
Agree 


□  3 


□ 


1:  Strongly 
Disagree 


Figure  21.  Participant  Ratings  for  IMPACT'S  Feedback  System  —  76%  of  participant 
ratings  across  the  five  measures  were  4  or  higher.  The  numbers  inside  the  bars  indicate  the 
number  of  participants  who  provided  a  rating  at  that  scale  value. 


Participant  comments  were  positive  in  regards  to  the  feedback  provided  by  IMPACT’S  FIAI, 
especially  the  Active  Play  Manager,  f 


which  participants  felt  helped  maintain 
their  situation  awareness  despite  having 
a  map  cluttered  with  six  UxV  routes. 
Participants  made  several 
recommendations  for  improving  the 


“When  the  map  was  real  complicated  and  busy 
the  Active  Play  Manager  made  it  real  easy  for 
me  to  go  through  and  tell  what  the  different 
pieces  were  doing.  "  -  UAV  Operator 

V.  J 


Active  Play  Manager  including  providing  play  status  information  instead  of  needing  to  drill 
down  to  the  Play  Status  Tile  (Figure  16)  and  using  it  to  center  the  map  on  plays. 


Test  Bed 

Finally,  participants  were  asked  to  rate  IMPACT’S  test  bed,  which  included  the  map,  screen  and 
interface  layout,  speech,  touch,  and  keyboard  and  mouse  input  modalities,  and  the  chat  system. 
Once  again,  generally  all  five  parameters  were  rated  positively  (see  Figure  22). 
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Agree  » 
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Disagree 


Figure  22.  Participant  Ratings  for  the  IMPACT  Test  Bed  —  93%  of  participant  ratings 
across  the  five  measures  were  4  or  higher.  The  numbers  inside  the  bars  indicate  the 
number  of  participants  who  provided  a  rating  at  that  scale  value. 


Almost  every  participant  mentioned  that 
the  map  became  cluttered  when  certain 
plays  (e.g.,  sector  search  -  see  Figure 
23)  were  called.  Several  participants 
suggested  that  it  was  difficult  to  connect 
what  they  were  doing  in  the  Play 

Creation  Tile  with  what  was  happening  on  the  map.  Participants  also  expressed  a  desire  to 
interact  directly  with  the  map  to  call  plays  by  dragging  UxV  icons  to  a  location. 


r 

“You  don  7  get  a  lot  of  focus,  kind  of  lose  track 
of  who  ’.s'  doing  what  and  what’s  doing  what.  ”  - 
Security  Force  Specialist 


Figure  23.  An  example  of  the  sandbox  display  being  cluttered  during  planning  for  a  sector 
search  play. 
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The  feedback  on  touch  and  speech  was  mixed;  in  general,  participants  seemed  to  like  the  idea  of 
being  able  to  execute  plays  via  touch  and  speech.  However,  participants  expressed  concerns 
about  the  touchscreen’s  calibration  and  lack  of  precision  (a  participant  might  touch  an  icon  three 
times  before  the  system  registered  it)  and  the  speech  system’s  poor  accuracy  (the  word  error  rate 
was  21.95%).  Objective  data  was  also  collected  on  the  modality  (mouse,  touch,  or  speech)  that 
participants  used  to  call  plays  during  the  mission  (when  participants  could  choose  the  modality). 
Though  several  participants  had  positive  comments  about  speech  and  touch,  participants  tended 
to  use  the  mouse  more  than  touch  or  speech  (see  Figure  24  -  note  that  speech  is  labeled 
speech/mouse  because  when  participants  used  speech  during  the  mission  they  always  used  it  in 
conjunction  with  the  mouse.  For  example,  a  participant  would  initiate  a  play  call  with  a  speech 
command  but  execute  the  play  by  clicking  the  checkmark  with  the  mouse  instead  of  saying 
“Confirm”  to  execute  the  play  by  speech  command).  In  fact,  only  one  participant  tried  to  use  the 
touchscreen  to  call  plays  during  the  mission  and  only  two  participants  tried  to  use  speech. 
Participants  also  made  a  higher  percentage  of  major  errors  (defined  as  failing  to  complete  a  play 
correctly)  when  using  touch  than  mouse  or  speech  (see  Figure  24). 


Plays  Attempted 
Major  Errors 


Mouse 

(6  Participants) 


Touch 

(1  Participant) 


'///////A 
Speech/Mouse 
CONFIRM 
(2  Participants) 


Figure  24.  Number  of  plays  attempted  and  number  of  major  errors  by  modality  type. 


Participants  were  also  faster  at  completing  plays  using  the  mouse  as  compared  to  using  touch  or 
speech  (see  Figure  25).  However,  this  difference  most  likely  reflected  the  specific  problems 
participants  had  with  the  touch  (not  precise  enough)  and  speech  (not  accurate  enough).  In  fact, 
when  only  correctly  completed  plays  (i.e.,  no  major  errors)  were  examined  the  difference 
between  time  to  complete  a  play  with  the  mouse  and  speech  was  only  2.5  seconds  (see  Figure  26 
-  note  that  participants  never  correctly  completed  a  play  using  touch). 
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Figure  25.  Time  to  Complete  a  Play  Call  by  Modality.  (Error  Bars  are  the  Standard  Errors 
of  the  Means) 


Figure  26.  Time  to  Complete  a  Correct  Play  Call  by  Modality.  (Error  Bars  are  the 
Standard  Errors  of  the  Means) 

Mission  Debrief 

Both  UAV  operators  and  security  force  personnel  had  positive  comments  about  IMPACT’S  real 
world  utility.  For  example,  one  UAV  operator  discussed  how  IMPACT  could  provide  a 
Combined  Air  Operations  Center  (CAOC)  commander  with  an  excellent  overview  of  the 
battlespace,  while  one  security  force  specialist  felt  that  IMPACT’S  capabilities  could  reduce  the 
number  of  manned  patrols  and  watch  towers  needed  during  base  defense  operations.  Participants 
also  suggested  that  the  scenario  could  be  improved  by  increasing  the  mission  length,  adding 
important  mission  considerations  like  fuel  (UxVs  had  unlimited  fuel),  airspace  deconfliction,  and 
communication  range  as  well  as  by  adding  more  complex  intruder  events  like  mortar  launches, 
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gate  runners,  vehicle -borne  improvised  explosive  devices,  and  base  perimeter  breaches.  Security 
force  personnel  provided  descriptions  of  base  defense  tasks  like  random  anti-terror  measures  that 
should  be  incorporated  in  future  scenarios  (e.g.,  randomly  searching  a  section  of  the  base’s 
perimeter  fence  for  signs  of  tampering),  cordons  (e.g.,  blocking  off  the  area  surrounding  a 
suspicious  vehicle),  and  blockades  (e.g.,  surrounding  a  suspicious  watercraft)  as  well  as 
constraints  (e.g.,  UGVs  should  always  have  at  least  50%  fuel  so  they  can  remain  effective  in  the 
case  of  an  unexpected  intruder). 
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5.  DISCUSSION 


The  goal  of  the  IMPACT  project  is  to  design  a  system  that  enables  one  operator  to  efficiently 
manage  multiple  heterogeneous  UxVs  in  dynamic  operating  environments.  To  this  end,  multiple 
autonomous  components  are  being  developed.  CCAs  rapidly  calculate  optimal  vehicle  routes 
based  on  situational  constraints.  This  involved  making  advancements  in  terms  of  developing 
motion  models  for  ground  and  surface  vehicles  and  trajectory  planning  for  new  plays.  Intelligent 
agents  recommend  plans  based  on  situational  factors  as  well  as  in  response  to  operator  requests. 
During  this  reporting  period,  knowledge  of  mission-related  activities  was  captured  in  the 
ontology  and  reasoning  was  enhanced  to  accommodate  actions  that  are  contextually  appropriate. 
The  Rainbow  autonomies  framework  monitors  the  ongoing  situation  and  alerts  the  operator  to 
errors  which  involved  modeling  plans  from  play  information  and  the  generation  of  gauges  from 
play  specifications  to  monitor  performance.  These  autonomous  tools,  along  with  advances  in 
novel  HAI,  provide  the  operator  with  the  ability  to  remain  agile  in  demanding  simulated  base 
defense  missions.  Another  significant  accomplishment  was  the  integration  of  these  technologies 
into  a  high-fidelity  simulation  featuring  the  Fusion  framework  and  this  required  developing  a 
message  standard  to  bridge  CCA,  IAs,  and  the  Rainbow  monitoring  service.  Besides  these 
development  efforts  that  improved  the  IMPACT  simulation,  other  related  research  efforts  were 
conducted  to  develop  a  model  of  an  operator’s  workload  as  well  as  determining  factors  that 
influence  the  transparency  of  IA’s  reasoning  to  the  operator.  Publications  related  to  the 
advancements  in  automation  are  provided  after  the  references. 

The  reported  study  examined  the  usability  of  the  IMPACT  Spiral  1  system.  Even  though  the 
Spiral  1  feature  set  was  a  subset  of  the  envisioned 
Spiral  2  IMPACT  system,  five  out  of  six  participants 
strongly  agreed  that  IMPACT  has  the  potential  to  be  a 
great  aid  in  future  UxV  operations.  Additionally,  all 
participants  agreed  that  IMPACT  has  the  potential  to 
improve  operator  situation  awareness  and  reduce 
operator  workload.  Participants  rated  both  the  overall 
IMPACT  system  and  system  subcomponents 
including  play  calling,  autonomy,  feedback,  and  testbed  positively. 


r  -\ 

“For  force  protection  this  would 
be  spectacular.  ”  -  UAV  Operator 

“It’s  exactly  what  you  need. 

They  ’re  great  tools.  ”  -  Security 
Force  Specialist 


This  study  also  examined  the  modality  that  participants  used  when  calling  plays.  Participants 
overwhelmingly  used  the  mouse  compared  to  the  touchscreen  or  speech  recognition,  and  were 
faster  and  more  accurate  with  the  mouse.  Several 
factors  may  have  contributed  to  these  results. 

Multiple  participants  had  difficulties  with  the 
touchscreen  registering  their  inputs.  For  example,  it 
would  often  take  a  participant  multiple  attempts  of 
touching  a  play  icon  before  the  system  responded.  In  fact,  some  participants  suggested  in  their 
comments  that  if  the  touchscreen  had  worked  better  they  would  have  been  more  likely  to  use  it. 
Several  participants  spoke  favorably  of  speech  in  their  comments,  especially  the  security  force 
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“ Touchscreen  could  be  extremely 
intuitive  and  quick  if  implemented 
correctly.  ”  -  UAV  Operator 


personnel,  who  mentioned  that  the  speech  commands  were  very  similar  to  the  dispatch  calls  they 
make  during  security  force  operations.  However,  this  preference  was  not  reflected  in 
performance,  as  participants  used  the  mouse/keyboard  to  call  plays  more  than  the  speech. 

Several  participants  commented  that  they  weren’t  completely  familiar  with  the  speech 
vocabulary,  suggesting  that  training  may  not  have  been  sufficient.  In  the  end,  participants  may 
have  chosen  to  use  mouse/keyboard  due  to  its  reliability;  clicking  a  play  icon  with  the  mouse 
consistently  resulted  in  the  desired  action,  while  touching  a  play  icon  or  issuing  a  voice 
command  often  failed  to  support  task  completion. 

Limitations 

The  biggest  limitation  of  this  study  was  the  lack  of  objective  measures;  though  participants 
provided  positive  subjective  feedback  in  regards  to  IMPACT,  the  extent  to  which  IMPACT 
improves  operator  performance  is  yet  to  be  determined.  The  mission  duration  was  also  short, 
limiting  the  opportunity  to  investigate  the  degree  to  which  operators  could  seamlessly  transition 
between  plays.  Additional  limitations  included  the  small  number  of  participants  and  the  length 
of  time  (only  -3.25  hours)  participants  were  exposed  to  IMPACT.  Operators  with  greater 
experience  with  IMPACT  may  have  been  more  comfortable  using  the  touchscreen  and/or  speech 
recognition. 

Towards  Spiral  2 

Participant  feedback  informed  and  improved  IMPACT’S  Spiral  2  development.  For  example, 
participants  expressed  a  desire  to  directly  manipulate  UxVs  and  call  plays  from  the  map,  features 
that  will  be  implemented  in  Spiral  2.  Participant  concern  about  map  clutter  led  to  plans  for  a 
decluttering  tool  that  provides  smart  automatic  decluttering  (e.g.,  dimming  all  unrelated  UxVs 
when  calling  a  play)  as  well  as  manual  controls  for  the  operator  to  declutter  situationally  relevant 
details  like  route  waypoints  and  route  labels.  Finally,  participant  suggestions  of  improving  the 
details  panel  led  to  the  development  of  play  widgets  that  allow  an  operator  to  graphically 
manipulate  loiter  patterns  (as  opposed  to  changing  textbox  values).  As  shown  in  Figure  26,  users 
can  change  loiter  direction  by  clicking  the  arrow  and  can  change  loiter  size  by  clicking  and 
dragging  the  loiter  pattern. 
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Figure  27.  Loiter  Pattern  Widget  that  enables  an  operator  to  quickly  change  loiter 
parameters  directly  from  the  sandbox  map,  such  as  loiter  direction,  radius,  and  position. 

Participant  feedback  also  generated  research  questions  that  led  to  additional  empirical  studies. 

For  example,  several  participants  felt  the  Play  Creator  tile  could  be  improved  by  organizing  play 
icons  by  vehicle  type  rather  than  by  play  type.  A  study  was  conducted  examining  the  effects  of 
icon  organization  and  the  results  supported  participant  opinions;  icons  organized  by  vehicle  type 
may  improve  an  operator’s  ability  to  locate  the  correct  icon  (Mersch,  Behymer,  Calhoun,  Ruff,  & 
Dewey,  2016).  Additionally,  participants  liked  teaming  with  the  IA,  but  expressed  a  desire  to 
understand  how  the  IA  reasoned.  To  support  this,  several  plan  comparison  tools  were  developed 
to  help  operators  compare  IA  generated  plans,  with  a  tool  using  a  plan  comparison  plot  showing 
promise  (Behymer,  Mersch,  Ruff,  Calhoun,  &  Spriggs,  2015). 

In  Spiral  2,  IMPACT  capabilities  will  be  significantly  enhanced  and  improved.  Speech 
recognition  will  be  more  accurate  and  the  list  of  commands  will  be  expanded.  Additionally,  a 
higher  touch  resolution  touchscreen  will  be  added  to  improve  the  IMPACT’S  touch  capability. 
New  plays  that  use  multiple  UxVs  in  a  highly  coordinated  fashion  will  be  developed,  such  as: 
communication  relays,  blockades,  cordons,  and  escorts.  Improvements  to  the  autonomous 
components  will  include  making  CCA  planned  routes  editable,  allowing  operators  to 
communicate  priorities  to  the  IA  to  improve  its  recommendations,  and  providing  more  status 
monitoring  capabilities  via  the  Rainbow  Autonomies  framework.  This  enhanced  system  will  be 
vital  as  the  difficulty  of  the  mission  will  be  increased  in  Spiral  2.  The  capabilities,  varieties,  and 
number  of  UxVs  under  the  operator’s  control  will  also  be  increased,  and  operators  will  be 
responsible  for  handing  vehicle  failures,  environmental  events,  and  base  intruders. 

In  order  to  assess  the  effectiveness  of  IMPACT’S  autonomous  capabilities,  the  Spiral  2  iteration 
of  IMPACT  will  be  compared  to  a  baseline  system  without  IAs,  Rainbow,  and  associated  HAI 
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components.  For  example,  in  the  baseline  conditions  participants  will  utilize  capabilities 
including  basic  “drag  and  drop”  movement  and  assignment  of  assets  to  set  patterns  (e.g.,  loiter), 
vector-directed  control,  and  waypoint-directed  plans  created  by  the  operator  via  waypoint 
manipulation  or  with  the  assistance  of  a  CCA  algorithm  rather  than  call  plays.  The  IMPACT 
system  will  have  the  baseline  capabilities  with  the  addition  of  an  expanded  CCA  to  assist  with 
route  planning,  IAs  to  support  plan  recommendations  and  monitoring  mission  goals,  and  the 
Rainbow  autonomies  framework  to  support  error  detection  and  improved  plan  recommendations. 
Operator  performance  and  overall  mission  effectiveness  is  expected  to  be  significantly  better 
with  IMPACT,  especially  for  more  complex  mission  conditions,  given  the  flexibility  and  agility 
it  affords  in  conducting  the  mission. 
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LIST  OF  ACRONYMS 


AFRL 

711HPW/RHCI 

AFSC 
AMASE 
ARPI 
ASD  R&E 
AVTAS 
CAOC 
CCA 
CTA 
ETE 
HAI 
HUMINT 

IMPACT 

IA 

IR 

MDARS 

ROZ 

SUS 

UAV 

UGV 

USV 


Supervisory  Control  and  Cognition  Branch,  Warfighter  Interface  Division, 

Air  Force  Research  Laboratory,  Airman  Systems  Directorate,  711th  Human 

Performance  Wing 

Air  Force  Specialty  Code 

AVTAS  Multi- Agent  Simulation  Environment 

Autonomy  Research  Pilot  Initiative 

Assistant  Secretary  of  Defense  for  Research  and  Engineering 

Aerospace  Vehicle  Technology  Assessment  and  Simulation 

Combined  Air  Operations  Center 

Cooperative  Control  Algorithm 

Cognitive  Task  Analysis 

Estimated  Time  Enroute 

Human  Autonomy  Interface 

Human  Intelligence 

Intelligent  Multi-UxV  Planner  with  Adaptive  Collaborative/Control 

Technologies 

Intelligent  Agent 

Infrared 

Mobile  Detection  Assessment  and  Response  System 

Restricted  Operating  Zone 

System  Usability  Scale 

Unmanned  Air  Vehicle 

Unmanned  Ground  Vehicle 

Unmanned  Surface  Vehicle 


UxV  Unmanned  Vehicle 
WPAFB  Wright-Patterson  Air  Force  Base 
XMPP  Extensible  Messaging  and  Presence  Protocol 
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APPENDIX  A  -  Background  Questionnaire 

Intelligent  Multi-UxV  Planner  with  Adaptive  Collaborative/Control  Technologies 

(IMPACT)  -  Initial  Evaluation 

ID# _ 


BACKGROUND  QUESTIONNAIRE: 


Experience  with  Unmanned  Vehicle  Operations  (ball  park  estimates): 


Vehicle  Name 

Non-Combat  Flight 
Hrs. 

Combat 

Flight  Hrs. 

Other  Previous  Flight  Experience  (Most  recent  first): 


Aircraft 

Ball  park  number  of  flight  hours 

Experience  with  Security  Forces  and/or  Base  Defense  (ball  park  estimates): 


J  ob/Deployment 

Length 
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APPENDIX  B  -  Overall  System  Questionnaire 

IMPACT  DEBRIEFING  QUESTIONNAIRE 

We  would  like  to  first  capture  your  general  impressions  of  the  overall  IMPACT  concept  for 
supporting  future  scenarios  involving  multiple  types  of  highly  autonomous  remotely  piloted 
vehicles  (UxVs).  The  IMPACT  concept  includes  all  the  display  formats,  symbology,  control 
methods  and  ability  to  call  plays  verbally  and  manually,  as  well  as  tailor  plan  parameters. 


1)  Please  indicate  which  response  best  matches  your  opinion  of  the  POTENTIAL  VALUE 
of  the  IMPACT  concept  for  future  UxVs  operations: 


Negative  Impact  G 

No  Aid  □ 

□ 

□ 

□ 

G  Great  Aid 

1 

2 

3 

4 

5 

2)  Please  indicate  which  response  best  matches  your  opinion  of  the  ability  of  the  IMPACT 
concept  to  aid  your  WORKLOAD  for  future  UxVs  operations: 


Negative  Impact  G 

No  Aid  G 

G 

G 

G 

G  Great  Aid 

1 

2 

3 

4 

5 
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3)  Please  indicate  which  response  best  matches  your  opinion  of  the  ability  of  the  IMPACT 
concept  to  aid  your  SITUATION  AWARENESS  during  future  UxVs  operations: 


Negative  Impact  G 

No  Aid  □ 

□ 

□ 

□ 

G  Great  Aid 

1 

2 

3 

4 

5 

Next  we  would  like  to  capture  your  general  impression  on  how  USABLE  you  found  this 
first  instantiation  of  IMPACT  in  general.  This  involves  responses  to  a  scale  and  open 
ended  questions. 
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System  Usability  Scale 
Page  1  of  2 

Please  answer  the  following  questions  by  circling  a  single  number 
from  1  (strongly  disagree)  to  5  (strongly  agree). 


Strongly 

Disagree 


Strongly 

Agree 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 


think  that  I  would  like  to  use  this  system  frequently. 

found  the  system  unnecessarily  complex. 

thought  the  system  was  easy  to  use. 

think  that  I  would  need  the  support  of  a  technical 
person  to  be  able  to  use  this  system. 

found  the  various  functions  in  this  system  were 
well  integrated. 

thought  there  was  too  much  inconsistency  in  this 
system. 

would  imagine  that  most  people  would  learn  to  use 
this  system  very  quickly. 

found  the  system  very  cumbersome  to  use. 

felt  very  confident  using  the  system. 


10.  I  needed  to  learn  a  lot  of  things  before  I  could  get 
going  with  this  system. 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


Please  answer  the  following  questions  with  your  honest  opinion, 
providing  as  much  detail  as  possible: 

1 1 .  What  did  you  like  most  about  the  system? 
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System  Usability  Scale 
Page  2  of  2 


1 2.  What  did  you  like  least  about  the  system? 


1 3.  What  was  confusing  about  the  system? 


1 4.  How  would  you  improve  the  system? 
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APPENDIX  C  -  Component  Specific  Questionnaires 

FEEDBACK  ON  IMPACT  FEATURES 

Next  we  are  going  to  have  you  consider  specific  features  of  the  IMPACT  concept,  thinking 
you  might  have  additional  comments  or  suggestions  that  you  haven’t  already  raised.  You 
may  want  use  the  IMPACT  simulation  or  our  handouts  for  reference  in  providing  your 
feedback.  For  some  features,  we  would  also  appreciate  you  providing  a  few  ratings. 

a.  One  key  feature  is  the  ability  to  direct  the  UxYs  with  HIGH-LEVEL  AUTOMATED 
PLAYS.  This  involved  several  components  listed  below.  Do  you  have  any  additional 
thoughts  of  what  you  liked  or  didn’t  like  about  any  of  these,  or  suggestions  for 
improvement? 

1.  Play  Tile:  Tile  with  icons  for  default/pre-planned  plays 

2.  Work-flow  tiles:  Expandable  tiles  to  view/specify  play  details 

3.  Preset  Panel:  size,  visibility,  optimize,  priority,  sensor  options;  icons 

4.  Steps  to  Call  High-Level  Plays:  verbal  and/or  manual  commands 

5.  Steps  to  Designate  a  Location:  voice,  “hand  tool”,  drop  down  menu 

6.  Steps  to  Seleet/De-seleet  Assets 

7.  Steps  to  Call  up  Alternative  Plans  from  Autonomy 

8.  Steps  to  Tailor/Change  Default  Parameters:  change  preset,  add  UxV,  change  loiter,  move  stare 
point,  modify  route,  add  sensor,  etc. 

9.  Steps  to  Tailor/Change  Parameters  of  Plays  in  Progress 

10.  Approval  Steps:  use  of  verbal  command  or  checkmark  selection  to  approve  plan  &  initiate  play 

Please  rate  your  view  of  the  POTENTIAL  VALUE  of  the  ability  to  call  HIGH-LEVEL 
AUTOMATED  PLAYS  for  future  UxVs  operations: 


Negative  Impact  D 

No  Aid  D 

□ 

□ 

□ 

O  Great  Aid 

1 

2 

3 

4 

5 

Comments: 


Also  use  the  scale  below  to  indicate  the  USABILITY  of  the  interface  to  call  HIGH-LEVEL 
AUTOMATED  PLAYS  (the  “system”  in  each  row  below)  for  future  UxVs  operations: 


Strongly  Strongly 

Disagree  Agree 


1 .  1  thought  the  system  was  easy  to  use. 

1 

2 

3 

4 

5 

2.  1  found  the  various  functions  in  this  system  were 
well  integrated. 

1 

2 

3 

4 

5 

3.  1  thought  there  was  too  much  inconsistency  in  this 
system. 

1 

2 

3 

4 

5 

4.  1  would  imagine  that  most  people  would  learn  to  use 
this  system  very  quickly. 

1 

2 

3 

4 

5 
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b.  Advances  in  AUTONOMY  also  plays  a  key  role  in  the  IMPACT.  Below  are  a  few 
features  to  prompt  additional  feedback  on.  Do  you  have  any  additional  thoughts  of 
what  you  liked  or  didn’t  like  about  any  of  these,  or  suggestions  for  improvement? 

1.  Route  Planning:  time  it  takes  to  generate  a  plan,  acceptability  of  plan,  etc. 

2.  Recommended  Assets:  blue  brackets  on  UxV  symbols 

3.  Suggested  Plays:  prompt  in  feedback  window,  button  options  to  ignore,  view,  or  accept 

4.  Quality  Graph  of  Play’s  Progress:  sprocket  pie  slice  size,  threshold  indicators,  color  warnings 

5.  Suggested  Plays/Actions:  messages  in  feedback  bar 

Please  rate  your  view  of  the  POTENTIAL  VALUE  of  the  role  of  AUTONOMY  in 
IMPACT: 


Negative  Impact  G 

No  Aid  □ 

□ 

□ 

□ 

G  Great  Aid 

1 

2 

3 

4 

5 

Comments: 


Also  use  the  scale  below  to  indicate  the  USABILITY  of  AUTONOMY  (the  “system”  in  each 
row  below)  as  illustrated  in  IMPACT  for  future  UxVs  operations: 


Strongly  Strongly 

Disagree  Agree 


1 .  1  thought  the  system  was  easy  to  use. 

1 

2 

3 

4 

5 

2.  1  found  the  various  functions  in  this  system  were 
well  integrated. 

1 

2 

3 

4 

5 

3.  1  thought  there  was  too  much  inconsistency  in  this 
system. 

1 

2 

3 

4 

5 

4.  1  would  imagine  that  most  people  would  learn  to  use 
this  system  very  quickly. 

1 

2 

3 

4 

5 
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c.  Providing  FEEDBACK  OF  STATUS  was  also  a  key  feature  in  IMPACT.  Do  you  have 
any  additional  thoughts  of  what  you  liked  or  didn’t  like  about  any  of  these,  or 
suggestions  for  improvement? 

1.  Table  of  Plays:  list  of  active  plays  &  any  warnings 

2.  Mini  Play  Tile:  plus,  line,  square  denoting  point,  line,  area  search  &  other  shape  coding 

3.  Detailed  Play  Tile:  play  icon,  name,  assets,  quality  sprocket,  edit  button 

4.  Feedback  on  Quick  Reaction  Force  state:  lightning  bolt  pixel  tag 

5.  Edit  Mode:  feedback  on  Play  Tiles  &  Workflow  tiles 

6.  Vehicle  Type:  shape  coding  for  air,  ground,  sea  UxVs 

7.  Detailed  Vehicle  Tile:  play  icon,  colors  for  play  status  &  fuel,  payload  info,  call  sign 

8.  Vehicles  on  Map:  shape  coding  &  indicators  on  selection  &  warning  status 

9.  Routes  on  Map:  dashed=proposed,  solid=active 

10.  Caution/Warning  Alerts:  play  tile,  vehicle  tile,  colors  on  vehicle  icons,  workflow  tile 

Please  rate  your  view  of  the  POTENTIAL  VALUE  of  the  role  of  FEEDBACK  OF  STATUS 
in  IMPACT: 


Negative  Impact  D 

No  Aid  □ 

□ 

□ 

□ 

D  Great  Aid 

1 

2 

3 

4 

5 

Comments: 


Also  use  the  scale  below  to  indicate  the  USABILITY  of  the  FEEDBACK  (the  “system”  in 
each  row  below)  as  illustrated  in  IMPACT  for  future  UxVs  operations: 


Strongly  Strongly 

Disagree  Agree 


1 .  1  thought  the  system  was  easy  to  use. 

1 

2 

3 

4 

5 

2.  1  found  the  various  functions  in  this  system  were 
well  integrated. 

1 

2 

3 

4 

5 

3.  1  thought  there  was  too  much  inconsistency  in  this 
system. 

1 

2 

3 

4 

5 

4.  1  would  imagine  that  most  people  would  learn  to  use 
this  system  very  quickly. 

1 

2 

3 

4 

5 
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d.  Finally,  the  TESTBED  is  a  key  element  to  IMPACT.  Do  you  have  any  additional 
thoughts  of  what  you  liked  or  didn’t  like  about  any  of  these,  or  suggestions  for 
improvement? 


1.  Tactical  Situation  Awareness  Display:  interactive  and  dynamic  map 

2.  Color  Coding:  all  routes/UxVs  on  same  play  a  unique  color;  unassigned  UxVs  gray 

3.  “Selected”  status:  white  (operator)  or  blue  (autonomy)  brackets;  indication  of  selected  UxV  on 
map  and  in  UxV  tile 

4.  Left  Side  Display:  chat  and  status  information 

5.  Right  Side  Display:  payload  sensor  info 

6.  Voice  Commands:  headset,  mic,  feedback  bar,  vocabulary  (including  UxV  type  distinctions) 

7.  Touch  Displays:  sensitivity,  reach  envelop,  feedback 

8.  Mouse/Keyboard:  implementation  of  push-to-talk 

9.  Moving/Managing  tiles:  was  it  necessary  to  move  tiles,  do  you  have  a  preference  on  tile 
locations  across  the  screens 

Please  rate  your  view  of  the  POTENTIAL  VALUE  of  the  role  of  TESTBED  in  IMPACT: 


Negative  Impact  D 

No  Aid  D 

□ 

□ 

□ 

D  Great  Aid 

1 

2 

3 

4 

5 

Comments: 


Also  use  the  scale  below  to  indicate  the  USABILITY  of  the  TESTBED  (the  “system”  in 
each  row  below)  as  illustrated  in  IMPACT  for  future  UxVs  operations: 


Strongly  Strongly 

Disagree  Agree 


1 .  1  thought  the  system  was  easy  to  use. 

1 

2 

3 

4 

5 

2.  1  found  the  various  functions  in  this  system  were 
well  integrated. 

1 

2 

3 

4 

5 

3.  1  thought  there  was  too  much  inconsistency  in  this 
system. 

1 

2 

3 

4 

5 

4.  1  would  imagine  that  most  people  would  learn  to  use 
this  system  very  quickly. 

1 

2 

3 

4 

5 
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APPENDIX  D  -  Post-Mission  Debrief  Guide 
Post-Mission  Debrief  Guide 

Note:  This  portion  of  the  post-mission  debrief  will  be  semi-structured.  While  all  question 
categories  will  be  covered,  the  specific  questions  may  vary  from  interview  to  interview  (e.g.,  base 
defense  personnel  will  be  asked  different  questions  than  UxV  operators ).  The  questions  listed 
are  provided  as  examples  of  the  types  of  questions  that  may  be  asked. 

1.  Air  Force  Specialty  Code  (AFSC)  related  example  questions 

a.  How  would  you  incorporate  unmanned  vehicles  with  these  capabilities  into  base 
defense? 

b.  What  challenges  do  you  envision  for  integrating  unmanned  vehicles  into  base  defense 
operations? 

c.  How  do  you  think  the  ability  to  utilize  different  types  of  unmanned  vehicles  and 
associated  advancements  in  autonomy  will  impact  base  defense  operations? 

d.  What  challenges  do  you  envision  for  managing  multiple  unmanned  vehicles? 

2.  Scenario  related  example  questions 

a.  What  was  challenging  about  the  scenario? 

b.  How  can  we  make  the  scenario  more  challenging? 

c.  If  you  had  unlimited  time  and  funds  to  design  a  training  scenario  to  train  base  defense 
personnel  (or  UxV  operators)  what  would  that  scenario  look  like?  How  would  you 
evaluate  trainee  performance? 

d.  What  mistakes  would  a  novice  base  defense  operator  make  in  this  scenario? 

e.  Is  scenario  representation  sufficient/adequate  to  assess  new  interface  approach? 

How  could  the  scenario  be  improved?  What  is  missing  from  the  scenario? 

f.  To  prioritize  future  developments,  what  other  types  of  tasks  and/or  information  feeds 
should  be  developed  and  integrated  into  the  simulation? 

3.  IMPACT  system  related  example  questions 

a.  How  would  this  interface  help  you  manage  the  challenge  of  managing  multiple 
unmanned  vehicles?  How  would  it  be  inadequate? 

b.  What  interface  features  did  you  find  most  useful? 

c.  What  interface  features  did  you  find  least  useful? 

d.  What  interface  features  did  you  find  confusing? 

e.  Would  you  prefer  the  autonomy  to  recommend  the  course  of  action  it  thinks  is  best  or 
to  generate  a  set  of  alternatives  you  could  select  from? 

f.  During  the  scenario  we  noticed  you  (e.g.,  only  used  voice  commands  instead  of  the 
other  input  modalities)?  What  made  you  (e.g.,  use  voice)? 
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